An estimate of Netflix's "fast lane" fee to Comcast -- $.86 per subscriber per month

Netflix, ISPs and transit providers blame each other for poor Internet performance. The public would be well served by transparency -- seeing the cost and traffic data underlying the debates on network neutrality and, more important, the high cost of US Internet service and lagging investment in US Internet infrastructure.

Generator Research has estimated the cost of delivering Netflix content in their report Over-the-Top Television, 2014, which includes an estimate of the cost to Netflix and Comcast of delivering Netflix content and of the fee Netflix is paying Comcast.

They begin by noting that Netflix's total cost of revenue (including content, delivery and other costs) was reported as $1,849 million in 2013 and they had an average of 29 million US subscribers for the year, then make the following assumptions:

• The cost of content is 80% of the total cost.
• The cost of delivery is 80% of the remaining cost.
• 75% of delivery cost is in the access network, 25% in the ISP network.
• Comcast earns $30 per month per subscriber, $24 of which is as a result of delivering Netflix traffic.
• 80% of Comcast traffic is due to Netflix.

Based on this, they estimate the total cost of delivering Netflix content as $2 per subscriber per month, with Netflix paying $.86 and Comcast $1.15. Assuming they split the cost in proportion to their monthly revenue ($7.99 for Netflix and $24 for Comcast), Generator guesses that Netflix's fee to Comcast is around $.28 per month per subscriber -- less than 5% pf the cost of a subscription. (Compare that to my ISP increasing my bill by 5-10% every year because, as a monopoly provider, they can).

Generator's conclusion is based upon several assumptions, which you may question, but it is a starting point in estimating cost of content delivery.

-----
Update 8/26/2014

Whatever the marginal cost, someone has to pay for investment in Internet infrastructure. Netflix CEO Reed Hastings says it would cost the ISPs very little to upgrade in order to provide the level of service they advertise to their customers.

It's worth noting that Netflix connects directly with hundreds of ISPs globally, and 99 percent of those agreements don't involve access fees. It is only a handful of the largest U.S. ISPs, which control the majority of consumer connections, demanding this toll. Why would more profitable, larger companies charge for connections and capacity that smaller companies provide for free? Because they can.

Hastings' post is one in a series on How to Save the Net running at Wired.com.

-----
Update 9/24/2014

Netflix CEO Reed Hastings was on a panel at the Cable and Telecommunications Association for Marketing EuroSummit Conference in Copenhagen last week. Mike Fries, CEO of Liberty Global was a fellow panelist. Hastings "jokingly" offered Fries the following deal:

Consumers are choosing Netflix and if we’re supposed to pay some of the cost of the network, maybe we should get some of the broadband revenue ... we’ll pay 10% of your network costs if we get 10% of broadband revenue. Or we’ll pay 10% of your network costs if you want to pay 10% of our content costs.

Hastings also stated that

The crazy thing in this whole debate is the actual amount of money being talked about is trivial to both of us – but we’re both worried on both sides about the precedent and what does it mean in the longterm?

Netflix blames Verizon, Verizon blames Netflix -- let's see the data.

Netflix says Verizon is a streaming bottleneck, Verizon says the congestion is upstream -- we need data not accusations.

Netflix recently showed messages like this one when they had to adjust the bit-rate on their streams to FIOS customers:

Verizon responded, claiming that the congestion was upstream from FIOS, and threatened legal action. Netflix says they will drop the messages for now.

The public and regulators need more data and transparency, not less. For example, here are Netflix data rate histories for Google Fiber, Comcast, Verizon FIOS and Verizon DSL:

This data clearly shows the improvement in Comcast speeds when they and Netflix agreed to terms in February. (One wonders how that rapid improvement was achieved and how much investment it required).

Verizon DSL is slow because of the technology. Those Verizon customers who can get FIOS are surely better off, but one wonders why Verizon fiber is so much slower than Google fiber and Comcast cable.

If Netflix congestion is upstream from Verizon, they should present data demonstrating that.

-----
Update 6/10/2014

Google also publishes data on YouTube streaming performance for ISPs. I checked the performance of Time Warner Cable, my not-so friendly ISP, in Los Angeles, my city.

The following graph shows YouTube video consumption for time slots beginning at 6 AM and peaking at 9 PM.

Google also reports the percent of the time an ISP delivers high definition (720p), standard definition (360p) and lower quality. For me, the worst performance, HD 81% of the time and SD 19%, occurs between 9-10 PM.

This is one more source of ISP performance data -- check it out and see how your ISP rates compared to others in your location (if there are others).

-----
Update 6/13/2014

The FCC will scrutinize the pay for service deals Netflix and Comcast and Verizon. They will also look at others, including Google.

FCC Chairman, Tom Wheeler, said he already had copies of the contracts, but I hope he asks the companies to provide traffic data and also disclose the cost of the infrastructure investments they make to keep service reasonable.

-----
Update 7/7/2014

YouTube has joined Netflix in linking to data when video delivery slows down. I was watching a YouTube video that began to stutter. When it paused to rebuffer, I saw this message:

When I clicked on the find out why link, it took me to Google's Video Playback Checklist, which lists things that might go wrong and suggests fixes -- no blame on your ISP -- so far. If none of those fixes help, they also provide a link to the Google Video Quality Report (described above). As we asked -- they show us the data.

-----
Update 7/19/2014

I've asked to see congestion data and to know the cost of alleviating the bandwidth shortage noted by Netflix customers and other Internet users. Mark Taylor, VP of Content and Media at Level 3, a transit provider used by Netflix, has given us some data and some costs. This diagram shows data at one of the of the ten exchange points between Verizon and Level 3:

We see that the congestion between Level 3's customer (Netflix) and a Verizon retail customer occurs in the forwarding of packets between their routers (in the same building). Both the Level 3 and Verizon networks are uncongested (green) but the links between their routers are congested (red).

That is the data for this exchange point, but what about the cost of alleviating the problem? Taylor says it is a few thousand dollars and five minutes work to install 10gb port cards in the routers. He says they have been asking Verizon to install the cards for many months and offers to have Level 3 pay for them. He says he will throw in the short cables between the routers.

If more capacity were needed after those eight port cards were installed, new equipment would have to be purchased, but that would be a small fraction of the cost of the networks the routers connect.

Note that the above diagram is based on a Verizon post describing the situation at the Los Angeles exchange point. Furthermore, it shows under utilization on the Verizon network -- no need for data caps in that case :-).

-----
Update 7/24/2014

David Young, Verizon's Vice President for Regulatory Affairs, has posted a reply to the claim by Mark Taylor of Level 3 that Verizon is causing Internet congestion. Young says the problem is not datacenter exchange point congestion, as Taylor claims, but congestion in the Level 3 network. In fact he says that:

Level 3 insists on only using its existing settlement-free peering links even though, as Level 3 surprisingly admits in their blog, these links are experiencing significant congestion.

This is Verizon's view of the situation -- transit providers like Level 3 are the problem:

But, when I looked back at Mark Taylor's post, I saw no such admission. In fact, Taylor says "our network has plenty of available capacity" and goes on to state that:

I can confirm once again that all of those thousands of links on the Level 3 network are managed carefully so that the peak utilizations look very similar to those Verizon show for their own network – IN BOTH DIRECTIONS (his caps).

These are conflicting statements -- who to believe?

Considering Verizon's track record, I'd give the benefit of the doubt to Level 3. Verizon promised to install FIOS fiber in my neighborhood several years ago, but subsequently changed their mind and decided instead on a "gentleman's agreement" dividing the wireless (Verizon) and land-line (cable companies) ISP markets. Indeed Verizon has a track record of broken promises -- Bruce Kushnick has documented 300 billion dollars worth of them.

That history makes me suspicious that Level 3 is telling the truth, but it is not proof. Proof would require transparency on both sides. Level 3 and Verizon should show us the data -- let's see the traffic logs for your networks. (Or at least let the FCC staff see them).

-----
Update 10/15/2014

I've asked to see the data. It's not congestion data, but Generator Research has estimated the cost of delivering Netflix content in their report Over-the-Top Television, 2014, which includes an estimate of the cost to Netflix and Comcast of delivering Netflix content and of the fee Netflix is paying Comcast.

Their estimate of Netflix's "fast lane" fee to Comcast is $.86 per subscriber per month. That is a small portion of the inflated monopoly/oligopoly prices we pay to Comcast and other Internet service providers. It seems to me that we should be paying more attention to the lack of competition in the ISP industry than to network neutrality.

-----
Update 11/9/2014

M-Lab monitors Internet performance and has published a report on ISP Interconnection and its Impact on Consumer Internet Performance.

The report concludes that

Using Measurement Lab data, and constraining our research to the United States, we observed sustained performance degradation experienced by customers of Access ISPs AT&T, Comcast, Centurylink, Time Warner Cable, and Verizon when their traffic passed over interconnections with Transit ISPs Cogent Communications (Cogent), Level 3 Communications (Level 3), and XO Communications (XO). In a large number of cases we observed similar patterns of performance degradation whenever and wherever specific pairs of Access/Transit ISPs interconnected. From this we conclude that ISP interconnection has a substantial impact on consumer internet performance -- sometimes a severely negative impact -- and that business relationships between ISPs, and not major technical problems, are at the root of the problems we observed.

It seems that interconnection between transit ISPs and ISPs that connect customers (access ISPs) is indeed the weak link in the content delivery chain.

The above figure gives an overview of download throughput between Access ISP and Transit ISP pairs during 2013. For each of the 28 graphs, the Y axis shows the % of hours with the given download throughput, and the X axis shows the download throughput (Mbps). Red, on the left, indicates sub-broadband speeds (download throughput below 4 Mbps). What this table demonstrates is not simply a proliferation of performance issues, but that these issues cannot be laid at the feet of any one Access ISP, or any one Transit ISP: no Access ISP performs badly to all Transit ISPs, and no Transit ISP performs badly for all Access ISPs. Therefore, if the problem is not at one end, and not at the other, it must be in the middle around the interconnection between the two.

For example, my ISP, Time Warner falls below 4 Mbps (red) over 30% of the time when delivering Cogent packets, but does much better with data from other transit ISPs.

The best video ever on network neutrality and the state of the Internet in the US

As long as analysts, geeks and lobbyists are the only folks who care about Internet infrastructure and policy, we can be pretty sure that nothing will improve, but what if the general public becomes interested?

Last night I watched episode five of the HBO comedy news cast "Last Week Tonight" in which host John Oliver narrated a 13 minute segment on network neutrality and the sorry state of the Internet in the United States.



I may be a biased geek and disgruntled consumer, but the segment was right on and, more important, extremely funny.

Is the Internet finally becoming a politically important issue? (Be sure to listen to the audience reaction at the end of the clip before you answer).

----
Update 6/26/2014

FCC Chairman Tom Wheeler responded to the video and Oliver followed up with another very funny segment.

------
Update 6/28/2014

Terry Gross interviewed John Oliver about his new HBO show -- the first 7 minutes are devoted to Oliver's Internet piece -- funny and informative.

---

Consumer groups and big corporations like Google and Facebook agreeing on net neutrality is like Lex Luther agreeing with Superman.

Comcast CEO Brian Roberts explained that there is no competition between Comcast and Time Warner Cable.

Picking Tom Wheeler to head the FCC is like hiring a dingo as your baby sitter.

President Obama golfing with Comcast CEO Brian Roberts

Network neutrality is a global issue

We talk about the Internet being designed as an end-to-end network in which network operators only make fast connections between computers -- the network does not provide services or differentiate between one person's packets and another's. This approach is generally favored by companies like Google and Skype, which offer Internet application services, and opposed by Internet service providers like Verizon and AT&T, which must deliver those services, and, in some cases, compete with them.

The US FCC recently endorsed the vision of an open, neutral Internet, and The Washington Post has published a story about Skype's lobbyists in Washington, who were quite happy with the FCC action. Skype is lobbying for neutral networks in every nation, not only the US. They feel the example set by the US FCC will have an impact on regulators in many other nations.

The US is among the world leaders in this policy, a nice place to be for a change.